In minimally invasive medical procedures (MIP), the doctor executes the clinical action of surgery or diagnosis based exclusively in the video acquired by a small endoscopic camera that is inserted in the targeted anatomical cavity. Such procedures are prone to errors and difficult to execute, with the surgeons having to undergo a long training period until mastering the surgical technique. In this context, improving visualization conditions and developing systems for assisting the doctor during the procedures is of importance to decrease clinical errors and to reduce the surgeon learning curve.
In most cameras, the relation between incident light and the image digitized values is nonlinear. Endoscopic cameras are no exception with such non-linearity being either due to limitations in camera/optics manufacturing, or used intentionally for compressing the spectral range of the sensors. The camera response function (CRF) models in part this non-linear relation by describing how (physically meaningful) incoming light is mapped to quantized image brightness values. Color characterization, i.e., estimation of the camera color mapping (CCM), also plays a big role in modelling a camera and needs to be accounted for to provide an accurate estimation of the CRF and to perform color standardization across cameras. To fully describe radiometrically for the camera, one needs to also account for vignetting. This effect is an attenuation of the light signal caused by the camera optics. It is present in many consumer cameras and more predominantly in MIP cameras and contributes to a poor image/video quality.
Calibration of the color model of the imaging device, comprising the CRF and the CCM, can be useful in many ways in the context of MIP. Since the color of organs and tissues is a cue in many diagnosis procedures, fidelity in color visualization is of key importance. Thus, the estimated camera model can be used in an image post-processing stage for the purpose of color constancy and white-balance, e.g., invariance of color to different illumination, lenses, and cameras. The CRF estimation may be a step for deblurring and is a crucial pre-processing step for the application of photometric vision algorithms, for example with the purpose of developing systems of computer-aided surgery (CAS). Reconstruction methods like shape-from-shading or photometric stereo assume a linear (or affine) mapping between physical scene radiance and image brightness, and such techniques are becoming increasingly popular in CAS as a way of performing 3D modelling of deformable surfaces. Vignetting compensation also has a major role in these applications as most existing algorithms for shape-from-shading or photometric stereo assume that no vignetting is present.
CRF estimation is a classical problem in computer vision with several different methods described in the literature. However, the camera characterization of medical endoscopes poses specific challenges that are not properly addressed by the current state-of-the-art. First, the anatomical cavity is illuminated via a light guide that runs along the endoscope, which means that we can neither consider a far light source, nor a light source coincident with the optical center. It is a situation of near-light source for which the vast majority of CRF estimation methods are not applicable. Moreover, the light source can hardly be considered punctual or isotropic (see FIG. 1). Thus, since the shape and format of the distal tip of the light guide varies across endoscopes, it is desirable to carry the characterization without making assumptions about illumination conditions. Second, since in MIP the lens scope is mounted on the camera-head immediately before starting the intervention, the calibration procedure should be fast, robust, and require minimal user intervention in order to be carried by the surgeon in the operating room (OR) without disturbing the existing clinical routine. It would also be advantageous to incorporate such camera characterization method with a method for geometric camera calibration that requires a single image of a checkerboard pattern acquired from a generic pose. There may be no similar solution for camera characterization that addresses this important usability requirement (as discussed below, single-image methods do not apply to medical endoscopes).